Data collection methods:
Administrative and spatial, observational, interviews, surveys
PSCI 2270 - Week 7
Department of Political Science, Vanderbilt University
October 10, 2023
Concepts vs. Indicators
Theories are made up of concepts (nodes):
- Inequality, civil discourse, media consumption, political knowledge, outgroup contact, views on immigration
- When creating a diagram of theory (DAGs) we took those for granted
Concepts are latent:
- We almost never observe “concepts”
- Instead we rely on “indicators” or “proxies”
Indicators are concrete:
- Concrete measure of a latent concept
- Sometimes they’re “good,” sometimes they’re “rough”
Sometimes there is slippage
Important to consider how do we construct indicators
- Some more straightforward: what is your age? how often do you watch TV?
- Others more complicated: political self-efficacy? racial discrimination?
- Have to create an operational definition of a concept to make it into a variable in our dataset
Sometimes there is slippage between latent concept and proxy, e.g.
- Responses to a specific policy question about affirmative action as a proxy for “racial resentment”
- Outcomes measured via self-reports may be clouded by social desirability bias (e.g., self-reported voter turnout)
Important to make measurement as unobtrusive as possible
Measurement Error
Reliability:
- Receive same answer over repeated measurements
- Individual measurement = exact value + chance error
- Chance errors tend to cancel out when we average across large sample
Validity:
- Avoid systematic errors, bias, in the same direction
- Individual measurement = exact value + chance error + bias
Data Collection Methods
Document analysis: Use of any audio, visual, or written materials as a source of data
Interview data: Data that are collected from responses to questions posed by the researcher to a respondent
Firsthand observation: Data that may be collected by making observations in a field study or in a laboratory setting
Everything is Possible
- Question: Which factors affect protest activity?
Protests in space
Geolocation and spatial data
- “Geographic Information Systems (GIS) in International Relations” by Jordan Branch
Geographic Information Systems (GIS) are being applied with increasing frequency, and with increasing sophistication, in international relations and in political science more generally. Their benefits have been impressive: analyses that simply would not have been possible without GIS are now being completed, and the spatial component of international politics—long considered central but rarely incorporated analytically— has been given new emphasis. However, new methods face new challenges, and to apply GIS successfully, two specific issues need to be addressed: measurement validity and selection bias. Both relate to the challenge of conceptualizing nonspatial phenomena with the spatial tools of GIS. Significant measurement error can occur when the concepts that are coded as spatial variables are not, in fact, validly measured by the default data structure of GIS, and selection bias can arise when GIS systematically excludes certain types of units. Because these potential problems are hidden by the technical details of the method, GIS data sets and analyses can sometimes appear to overcome these challenges when, in fact, they fail to do so. Once these issues come to light, however, potential solutions become apparent—including some in existing applications in international relations and in other fields.
What is GIS?
- Geographic Information System (GIS): Any system for the collection and analysis of data that are coded spatially (by location). Generally involves the use of a GIS software package for the creation and analysis of spatial data.
Vector data: Points, lines, and polygons to describe spatial features: a point for a feature at a single location, a line for a linear feature such as a road, or a polygon for a feature that covers a definable spatial area.
Raster data: Pixels, predefined equivalent-sized units that are then assigned a value for a single variable across the entire area covered by the data.
Raster vs vector data
Examples
Stasavage, David. 2011. States of Credit: Size, Power, and the Development of European Polities. Princeton, NJ: Princeton University Press.
- Shape or scale of polity/country matters
Starr, Harvey. 2013. On Geopolitics: Space, Place, and International Relations. Boulder, CO: Paradigm.
- Shared borders and interaction across them
Cederman, Lars-Erik, Kristian Skrede Gleditsch, and Halvard Buhaug. 2013. Inequality, Grievances, and Civil War. New York: Cambridge University Press.
- Location and size of ethnic or rebel groups
- What else?
Issues
Measurement validity:
- Changes in meaning of boundaries over time
- Changes in meaning of boundaries across space
Selection bias:
- Exclusion of units based on inability to code them geographically
Discussion
What kind of geolocation or spatial data can we use to study determinants of protest activity?
Protests in text
Text as data
- “Large-Scale Computerized Text Analysis in Political Science: Opportunities and Challenges” by John Wilkerson and Andreu Casas
Text has always been an important data source in political science. What has changed in recent years is the feasibility of investigating large amounts of text quantitatively. The internet provides political scientists with more data than their mentors could have imagined, and the research community is providing accessible text analysis software packages, along with training and support. As a result, text-as-data research is becoming mainstream in political science. Scholars are tapping new data sources, they are employing more diverse methods, and they are becoming critical consumers of findings based on those methods. In this article, we first describe the four stages of a typical text-as-data project. We then review recent political science applications and explore one important methodological challenge—topic model instability—in greater detail.
Four stages of text analysis
Obtaining text
- web-scraping vs crowdsourcing
From text to data
- supervised vs unsupervised methods
Quantitative analysis of text
- analysis of counts vs predictions with machine learning
Evaluating performance
- gold standard vs out-of-sample validation
Four uses of text data
Classification: Unsupervised machine learning methods compare the similarity of documents based on co-occurring features
Scaling: Use texts to locate political actors on ideological space
Text Reuse: Explicitly value word sequencing in judging document similarity
Natural Language Processing: Moving from “whom?” to “who did what to whom?”
Issues
Measurement reliability:
- Unsupervised Machine Learning produces different results
- Also is an issue with Large Language Models (e.g. ChatGPT)
Measurement validity:
- Unsupervised Machine Learning and Large Language Models are black boxes
Selection bias:
- We often do not have access to the full universe of documents (e.g. API restrictions)
Discussion
What kind of text data can we use to study determinants of protest activity?